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ABSTRACT 

This volume reports an effort to use the video media 
as an approach for the preparation of a battery of symbolic tests 
that would be empirically valid substitutes for criterion referenced 
Job Task Performance Tests. The graphic symbolic tests reguire the 
storage of a large amount of pictorial information vhich must be 
searched rapidly for display. At the time this video effort tras 
started, no completely satisfactory way had been fonnd for rapidly 
searching and displaying such information. The anticipated results 
that the video media vould provide satisfactory and economical 
solutions to these problems did not materialize. From a testing point 
of view, video symbolic tests place the test subject in a passive 
evaluative role of vatching someone else perform each task. Success 
in this passive role does not ensure his success in the active role 
of performing the same task. Also, the development costs of the video 
tests proved to be very high in terms of video equipment and test 
development time. To obtain video materials of acceptab2.e quality, 
vould require both quality video equipment and studio production 
facilities. As a result, it was recommended that video should not be 
further considered as a testing medium for performance analogues and 
that future efforts should be aimed at improving and refining graphic 
symbolic substitute tests. (Author/DEP) 
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SUMMARY 



Pioblem 

In previous studies, reported in Volumes II and III of this report, both criterion-referenced Job Task 
Performance Tests (JTPT) and matching graphic symbolic substitute tests were developed for measuring 
technicians' ability to perform electronic maintenance tasks. The symbolic substitute tests, as developed, 
employed numerous and cumbersome paper displays of equipment, test point readings and schematics as 
the mechanism for simulating the actual equipment conditions. While promising results were generally 
obtained, \he tests were expensive and required considerable preparation time. In addition, the 
trouble-shooting tests were the least satisfactory, in always requiring a highly skOled test administrator. 
This effort was an attempt to "'short circuit** these problems that must be solved for graphic type symbolic 
tests by using a video media. 

Approach 

The feasibility of using video tape recoidings was considered as a method for developing and 
administering symbolic performance tests for electronic technicians. Video tape recordings were used to 
document and analyze various types of job performance on an electronic subsystem. These included 
component removal and replacement, system checkout, special and general test equipment usage, 
adjustrtent and alignment, general tool usage, and troubleshooting. Video taped symbolic tests were then 
produced taking particular advantage of the media's capability to capture dynamic displays such as flashing 
lights and oscillating needles. 

Results 

A validation study was performed at Altus AFB, Oklahoma, using both graphic and video tape 
versions of performance tests. The video tape tests did not provide any increase in efflciency over the paper 
and pencil tests and appeared to have many technical problems associated with use of the video equipment 
itself. Video tests place the test subject in a passive, evaluative role- that of watching someone else 
perform the job task. Success in this passive role does not ensure his success in the active role of performing 
the task. This is not an appropriate role for an electronic technician. The cost of developing video symbolic 
tests is very high in terms cf video equipment and test development personnel time. The development of 
video tests in an operational environment presents production problems that limit the quabty of the 
resolution to an unacceptable level. As a result, *he development and administration of these video 
symbolic tests never advanced to the point of a systematic comparison with job task performance criterion 
tests using electronic technicians as subjects. 

Conclusions 

This effort surfaced the above-mentioned and other expensive developmental and administrative 
problems and solved few if any of the problems associated with the graphic type symbolic substitute tests. 
It was concluded that the video medium did not show sufflcient promise to warrant further development at 
this time. Future efforts concerning symbolic substitute testing should be aimed at improving and refining 
graphic symbolic substitute tests in keeping with the recommendations presented in Volume III of this series 
of reports. 
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PREFACE 



This report represents a portion of the Exploratoiy Development program of the 
Advanced Systems Division of the Air Force Human Resources Laboratory. This report 
W3S prepared by URS/Matrix Research Company, FaDs Church, Virginia 22042 under 
Contract F33615-71C-1505. Dr Edgar L. Shriver was the principal Investigator of the 
Matrix work. 

This document is the fourth of four volumes to be published concerning the 
evaluation of maintenance performance. Volumes I, II, and III are: 

1. Evaluating Maintenance Performance: An Analysis. AFHRLpTR-74-57 (I) in 

press. 

2. Evaluating Maintenance Performance: The Development and Tryout of Criterion 
Referenced Job Task Performance Tests for Electronic Maintenance, AFHRL-TR-74-57 
(II) Part I, in press. 

3. Evaluating Maintenance Performance: Test Administrator's Manual and Test 
Subject's Instructions for Criterion Referenced Job Task Performance Tests for 
Electronic Maintenance. AFHRL-TR-74-57 (10 Pirt II, in press. 

4 . EvaluatingMaintenance Performance: The Dcvelopmerit of Graphic Symbolic 
Substitutes for Criterion Referenced Job Task Performance Tests of Electrofflc 
Mamtenance. AFHRL-TR.74-57 (III), in press. 

The preparation of these volumes has been documented under Task 1710 10, 
Evaluatmg the performance of Air Force Operators and Technicians of Project 1710, 
Trammg for Advanced Air Force Systems. The effort represented by this volume was 
Identified as work unit 1710-10^5. Dr John P. Foley was task scientist. Mr John K. 
Klesch of the Advanced Systems Division was the contractor monitor. Dr Ross L. Morcan 
was the project scientist. 
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EVALUATING MAINTENANCE PERFORMANCE: 



A VIDEO AITROACH TO SYMBOLIC TESTING OF 
ELECTRONICS MAINTENANCE TASKS 



I. BACKGROUND AND RATIONALE 

The purpose oi this study was to continue the 
examination of methods for simulating the elec- 
tronics maintenance task as a means of measuring 
the proficiency of individual technicians. In 
previous studies, reported in Volumes II and III of 
this series of reports, a standard set of perform- 
ance tests was developed for a typical electronic 
subsystem, a doppler radar, the AN/AFN-147 and 
its computer, the AN/ASN-35. These tests were 
developed in two formats: actual criterion refer- 
enced job task performance tests (JTPT), and 
graphic symbolic representations of those tests so 
that performance capabflity c^uld be measured 
without requiring extensive electronic equipment 
support. The latter employed numerous paper 
displays of equipment, test point readings, and 
schematics as the mechanism for simulating the 
actual equipment conditions. While a positive 
relationship was obtained between individual 
performances on the two forms of the tests, the 
area of troubleshooting testing, which is the heart 
of the electronic technician's job , to date, has been 
the least satisfactory from an administrative stand- 
point. These symbolic troubleshooting tests 
provided a display of the test point reading (e.g., a 
waveform, or voltmeter display) for a wide range 
of equipment test points. These readings were 
those actually present in the set when a particular 
malfunction was inserted into the system. Two 
main difficulties were encountered in utilizing this 
technique. First, there was a requirement for a 
highly knowledgeable Test Administrator who 
could quickly And the display for a given test 
point when specified by the test subject; who 
could recognize when a requested item of informa- 
'tion was outside the boundaries of relevance to the 
problem and give a standard reply (i.e., "Those 
values are within tolerance"); and who could 
change symptom displays when an action had been 
specified that would cause a change in system 
values. In effect, the electronic equipment was 
replaced by the Test Administrator who had to 
respond just as the equipment would respond. 

A mord general shortcoming affecting all the 
symbolic tests was the lack of dynamk: response of 
the graphk displays. Many electronic indicators 
consist of movement of dials or needles, or other 
dynamic indicators such as flashing li^ts. Events 



such as these that occur across a temporal span 
could only be approximated or described in verbal 
terms to the technician when employing graphic 
techniques. 

To attempt to overcome these difficulties and 
shortcomings, an examination of the utility of 
other media as the display mechanism was under- 
taken. While there are a variety of such media that 
could be employed, video tape recording was 
selected since it offered the most apparent flexi- 
bility for capturing and viewing data in the most 
expeditious manner for research purposes. This 
lexibility meant that it could also simulate other 

•dia such as filmstrips or still slides without 
involving the reproduction time and expense 
associated with those media. 

The primary tasks of electronic maintenance 
that had been selected for study in this series of 
projects were: 

1. Component Removal and Replacement, 
dealing with all levels of mechanical assembly and 
disassembly from dust covers to plug-in compo- 
nents. 

2. System Checkout, which covers the standard 
checkout procedures used to verify system 
functioning at a gross level. 

3. Special and General Test Equipment Usage, 
dealing with the employment of the various types 
of testing equipment used in electronic mainte- 
nance. 

4. Alignment and Adjustment, dealing with the 
specified routines required to jring the electronic 
parameters of a system to the correct operating 
levels. 

5. General Tool Usage, dealing with the 
principle hand tools of the electronic trade, prima- 
rily the soldering tools. 

6. Troubleshooting, dealing with the ability of 
the technician to isolate and correct representative 
faults in the subject electronic system. 

In deciding how to employ video techniques to 
improve the graphic symbolic techniques pre- 
viously utilized, consideration was given to the 
deficiencies of the graphic format and how video 
capabilities could be used to overcome them. 
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In classic testing theory, elements of a total set 
of skills or knowledges are selected for measure- 
ment. In criterion referenced performance testing, 
* as represented in the previously developed JTPT 
for electronic maintenance, the totality of skflls 
and knowledges for producing selected outputs 
must/be employed to produce the desired output. 
If the output produced is within tolerance, this is 
accepted as evidence that a proper set of skills and 
^owlcdges has been exhibited in order to produce 
the output. In fact, by defiaition, this is an 
acceptable set of skflls and knowledges. It is 
interesting to note that this "set" represents a 
"chain** which leads to the production of the 
correct output. It is a "complete set" for this 
output. The elements of this "set** do not need to 
be tested individually to determine that they have 
been performed correctly. The evidence of a 
••within tolerance" output is sufficient to indicate 
that all the elements of the performance were 
"correct". This would seen to simplify testing 
procedures. But unfortunately it does not reduce 
the testing time required - or the equipment 
requirements. The time and equipment required 
for testing is stfll high even though the measure- 
ment of the product does not require much time. 

Is there anything to be gained by sampling a 
few of the skflls and knowledges within a "set" or 
"chain" which leads to a within tderance output? 
It would appear to reduce the time and equipment 
requirements. But what assumptions would be 
involved? One assumption is that selected elements 
from this set would bear a relationship to the total 
set. That is, faflure on an element would be 
indicative of faflure on the total set. This appears 
to be a reasonable assumption. A second assump- 
tion is that elements of a set would be meaningful 
to the subject outside the context of that set. This 
assumption appears difficult to meet in some 
cases. For instance, a pointer on a dial has no 
absolute meaning outside the context of a "set" 
leading to some purposeful end product. A 
subject's abflity to read the dial could be meas- 
ured. But an interpretation as to whether that 
particular reading was right or wrong cannot be 
made out of context. So this assumption could be 
met for the "skfll-knowledge" of reading a dial, 
but not for interpreting that reading. Likewise, the 
subject would not be able to select the proper dials 
or controls without the context supplied by a 
purpose: the purpose being the product of a 
certain output. Nor would the sequence of selec- 
tion of control actions and indicator interpreta- 
tions be meaningful without a stated purpose. 



Tnese things that cannot be measured out of 
context are the essential elements to be measured 
to determine if a subject can produce the desired 
system outputs. The things that can be measured 
are necessary but not r-ufiicient for producing the 
outputs. So it seems that in situations where the 
purpose of a set of skflls and knowledges is to 
produce some output, those skflls and knowledges 
cannot be measured independently of the context 
provided by that purpose. 

How does this analysis impact a test in which 
the subject would be observing and judging 
someone performing a "set" intended to produce 
the desired output? There is an advantage to such 
a test in that the standard performance can be 
captured and presented realistically on video. This 
would eliminate the requirement for equipment 
for every subject, and groups of subjects could be 
tested, resulting in time savings using this proce- 
dure. Ihe procedure also seems to overcome the 
problem of context. But there is a question of 
whether observing and critiquing someone else's 
performance taps the same skills and knowledges 
as does the subject's own performance. The answer 
to this must come fron an empircal study, and 
that, in turn, requires adequate testing materials. 

To guide the development of the initial set of 
video testing materials to examine this question, 
then, a general rationale was developed for each of 
the functional testing areas. This rationale is 
summarized in Table h 



IL Vi:>bO TECHNIQUE DESCRIPTION 
Approach 

The video equipment utflized to develop the 
testing materials consisted of one camera with 
viewfinder, a video tape recorder, one monitor, 
and several types of lenses. An Air Force organiza- 
tional maintenance shop (electronic) was the 
setting for fflming the maintenance tasks. A 
number of difficulties were encountered in using 
such a location. The benefits, however, did 
outweight the disadvantages. One advantage was 
direct access to all of the prime and test equip- 
ment, and to a sample of experienced technicians. 
Another advantage was access to a source of 
technical expertise pertaining to the prime equip- 
ment. It was necessary to work during times of 
relative inactivity, such as during the evening shift, 
to get adequate uninterrupted access to the equip- 
ment. In developing the test materials, typically, a 
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Table L Video Test Development Rationale 


Ttst Catc9Dry 


^ptr SymbDiicf 


VidtD symbolics 
Ttstin9 Approach 


Removal and 
Replacement 


S must identify equipment, locate module and 
component, and indicate fasteners involved in 
gaining access. 


1 . S must discriminate from among 
several removals which was the 
correct one. 

OR 

2. S must properly identify com* 
ponents as they are removed. 


Checkout 


S must select from among graphic equipment 
depictions which are required items, draw 
connections to correct points and indicate 
controls to be set to which positions. 


S views technician making bench set- 
up. Indicates on answer sheet 
erroneous missing connectbns. 

S observes technician making checkout 
and gives his interpretation of 
whether equipment is functioning 
properly. 


Special and 
Geneial Equipment 
Usage 


Given a front panel display of an item of 
test equipment and a specific problem, S 
must mark the controls and settings, and 
give the resultant readings. 


S views technician using item of 
test equipment in work context. 
Must identify any procedural 
errors, and interpret results 
obtained. 



Alignment/ 
Adjustment 



Give a series of graphic depictions, S must 
identify correct equipment to be used, 
location of adjustment point, critical values, 
and how adjustment is accomplished. 



S will be given an adjustment prob- 
lem and then given time to study 
his technical data. He wOl then 
be shown a series of test equip- 
ments from whkh he must identify 
that which is to be used. Finally, 
he will observe a technkian per- 
forming the adjustment and be 
required to identify errors made. 



General Tool 
Usage 



S must identify correct worksmanship on 
soldering task. Must also correctly locate 
and identify piece/parts for replacement 
according to schematic directions. 



S wfll view a piece/part replace- 
ment task being performed. He wfll 
be required to note any enors made 
In performance and evaluate resul- 
tant solder connections. 



Trouble- 
shooting 



S is told that there is a malfunction that 
he must find. He then can ask for any 
information he wants. In terms of test 
readings and values existing in the system. 
He must correctly identify the bad component. 



S wfll view a technician corulucting 
a troubleshooting effort. He wfll 
be shown the results of each check 
that is made, up to a given point. 
He will then be required to define 
either: 

a. the next steps to be taken 



b. the faulty component 



S will be informed as to what problem 
exists in the equipment. He wfll 
then observe the troubleshooting 
actions of the technician and iden- 
tify any incorrect actions. 
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technician was placed at the disposal of the test 
developers for the length of a shift. 

At the cutset of the data gathering, a technician 
was given a general assignment such as "Perform 
the XYZ adjustment" and an attempt would be 
made to film his actions. This produced a lot of 
dead time on the tapes in which notliing was 
happening as the technician studied his technical 
documentation, hunted for a piece of equipment, 
or thought about the problem. While these are all 
legitimate activities, they are not good video 
material. In addition, with only one camera, the 
test developers were seldom properly positioned to 
catch clearly two successive actions. While they 
could watch, for example, the technician's actions 
on top of the equipment, he might then shift to 
the back or side where his hands would be out of 
view. This quickly led to the necessity of stopping 
his actions while the camera was repositioned. The 
**non-interference" concept soon proved infea- 
sible with limited video equipment. Once the test 
developers started controlling the techr^'cian's 
actions to the extent of slowing him down or 
stopping him, it was a short step to "directing" the 
technician's actions. This ultimately proved to be 
the most efficient mode of operation. 

The approach generally used after the initial 
period of experimentation was to define the test 
structure and the action sequences required to 
present the situation, and then to stage the se- 
quences and have the technician perform defined 
actions. 

One of the major factors affecting the develop- 
ment of video test< is the procedure to be used in 
administering the tests. The format of materials on 
the tape must be governed by how active a role is 
to be taken by the Test Administrator (TA). 
Essentially there are three levels ai which the TA 
can be employed: full control, limited control, an<l 
self*administered tests. 

Under full TA control, the TA conducts the 
v^hole testing operation. He operates the recorder, 
provides narration, presents the questions, and 
controls the rate of video material presentation. 
This means that he must be completely familiar 
witihthevideo material and the testing concept so 
that he can properly program the adminLst-'-tion. 
It minimizes the pre-programming required on the 
video tape. 

Alternatively, the TA may be used in a minor 
role in administration, only to pass out forms, 
answer questions and start and stop the video tape 
on cue. In this mode, the narration and test 



instructions must be on the tape as well as the 
signals to the TA for recorder starting and 
stopping. This requires careful production, 
planning and timing of the video and audio 
material, which entails considerable editing and 
transferring of material from tape to tape. 

Finally, the goal may be to achieve a totally 
self-administered test. This may be technically 
feasible with the advent of cartridge and cassette 
recorders which a technician could easily load 
himself. There are two elements, however, which 
make this mode difficult to achieve. First, a tech- 
nician must be given time to think and analyze a 
situation once it has be'^n presented to him. 
Troubleshooting is not a matter of recalling 
specific answers or bits of information in response 
to a question. Secondly, th^ technician must be 
allowed to use his standard technical documen- 
tation. No one is expected or even permitted to 
troubleshoot without refening to the manuals on 
the equipment. Both of these activities, however, 
require time. This means that the tape recorder has 
to be shut down for a given period so that this 
work can be done. Further, each technician being 
tested would require varying amounts of time for 
reference and analysis, so that any pre-pro- 
grammed amount of time would be inappropriate 
for some percentage of the test subjects. While in a 
self-administered test the technician could stop the 
tape (or even reverse it to view again), this would 
not be appropriate for group administration, 
which is one of the goals of video testing. A 
recorder would have to be obtained that could be 
progranmied to start and stop automatically at 
specified pol: 

The issue of time necessary for technician 
review and analysis also impacts the previous 
alternative of a limited TA role. There is an easier 
solution available in that case,' however, in having 
the TA control the review time. Since that alter- 
native represented both challenge and feasibility, 
the video tef>t materials were developed based 
upon a limited TArole. 

In addition to the extent of the TA role in 
video testing, another consideration affecting test 
strategy is th<* choice between providing very 
structured response possibilities (e.g., multiple 
choice responses), or requiring constructed 
responses based on information presented. Since 
there was no clear rationale available for pre-selec- 
ting one approach or the other, both were 
employed. Under the first approach, questions 
were constructed that had the technician select, 
based on what he had been shown on the video 
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tape, the proper component name, fault, or next 
action required from among four choices. As 
designed, the technician had a card on which he 
punched out his selected response number after 
listening to the question and the response choices 
given verbally on the tape. An alternative approach 
to this would be to provide the questions in 
printed form for him to read. The trade-offs 
involved are that in the written question format 
the subject's visual attention is divided between 
the screen and the paper. In the verbally presented 
questions, the technician must remember the 
questions and alternative answers long enough to 
decide which one he thinks is correct. While the 
latter approach reduces the amount of reading 
required, it does impose a different source of 
Variance in the retention requirement. 

When video is used as the primary means of test 
administration, there are only a limited number of 
testing strategies available. The most compelling of 
these, based on the nature of the medium, is that 
of having the subject critique the performance 
demonstrated to him. In this, the subject observes 
the action of the demonstrator and then is asked 
questions concerning both the appropriateness of 
the actions shown and the correctness .vith which 
they were performed. The critiquing type activity 
is not, of course, the standard activity of the 
electronic technician. Certainly the question of 
whether success in such activities corresponds with 
the capability of performing maintenance activities 
on one's own would have to be demonstrated, 
since there is no basis for making such an 
assumption. The only case where critiquing 
activity does correspond to actual job duties is 
that of the supervisor. It may be that the evalu- 
ation of supervisory personnel is the most valid 
application of video tests using this strategy. 

Another basic testing approach available is that 
of having the subject state what he consider.^ to be 
the next required aciion based on given infor- 
mation. In th*.s, a subject might be shown a series 
of checkouts and their results and be asked what 
should be checked or replaced next. A recurrent 
problem in using this strategy in electronic 
maintenance is the presence of multiple correct 
answers. In terms of process, different technicians 
may proceed in different directions to arrive at 
a problem solution. Thus, the grading of responses 
to such questions become difficult. 

A third strategy is to use video to present 
discrete results for direct interpretation. For 
example, in testing the ability to use a voltmeter 
or oscilloscope the meter readings or waveforms 
are shown along with the control settings for the 



subject to read. The subject is asked specific 
questions concerning the instrument reading. The 
video medirjn does not permit him to operate the 
dials or, in itself, let him indicate what should be 
operated or set. 

To achieve any flexibility or expansion in the 
testing strategies avaHable, it is necessary to 
introduce other media uto the testing structure. 
The use of symbolic answer forms, for example, 
permit a recall type of response rather than the 
rec<^ 5 required in the video tests, 

lie o» . other media are introduced, the 

roie ci .ivleo becomes reduced and the flexible part 
of the testing system becomes the alternative 
medium. When this occurs, it is hard to rationalize 
the use of video as the presentation medium since 
lesG expensive alternatives come easily to mind. 

Description of the Video Test Materials 

Removal and Replacement. The tests in this 
area consisted of having a demonstrator, being 
given a specific assignment (e.g., "remove the 
crystal oscillator") proceed to remove a given 
component. For each instruction sevt J sequences 
were taped, consisting of the location and removal 
of the specified component and several sequences 
showing the removzd of the wrong component. 
The test subject would be shown any one of the 
sequences and asked whether it was correct and 
whether proper tools and procedures had been 
utilized. The main element of this testing was 
recognition and identification capability on the 
part of fhe test subject. 

The video taped materials of the removal/ 
replacement process permitted adaptation to 
eitfier of the original testing strategies outlined, 
that of recognition or that of identifying compo- 
nents as they were encountered in the taped 
sequence. 

System Checkout. The checkout procedures for 
the AN/APN-i47 Radar system involve making 
equipment connections with a test harness, 
generally mounted in the workbench, energizing 
the system, and then observing the response of 
several indicators as the input signals are varied. In 
the video testing of this task the procedures for 
setting up the equipment were taped and were to 
be tested one time. The checkout procedure itself, 
ho>vever, is an integral part of the troubleshooting 
process, so that it is essentially tested in every 
troubleshooting problem. The graded portion of 
the checkout test, then, was included in the 
troubleshooting tests. 
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The checkout procedure was tested by taping 
the results of the checkout with each of the 
various faults inserted into the radar system. This 
produced tapes with varied results to be inter- 
preted during the checkout. There were several 
possible conclusions that a technician could reach 
as a result of watching the checkout procedures: 

a. The radar system is functioning normally 
(within tolerances). 

b. The radar system is malfunctioning; suspect 
a problem in the frequency tracker.. 

c. The radar system is malfunctioning; suspect 
a problem in the receiver/transmitter. 

d. The radar system is malfunctioning; suspect 
a problem in the computer. 

e. The radar system is malfunctioning; suspect 
a problem in the antenna. 

The checkout procedure was considered 
particularly well adapted to the video approach 
since the movement of dial and meter indicators 
had to be observed over a specified time period. 
The re^onse of those indicators provides the cues 
to the technician as to the most likely general 
location of a specific malfunction. 

Special and Genera! Test Equipment Usage. In 
the use of test equipment the prime requisites are 
selection of the proper equipment, correct hook- 
up of the equipment, proper usage, correct reading 
of the resuhs, and, finally, correct interpietation 
of the results. 

For the special test equipment these elements 
were tested in the context of their application to 
the prime equipment since, by definition, that was 
their only application. The video tape for these 
tests showed a technician performing a defined 
function which required the use of the item of test 
equipment under test. The test subject had to 
verify whether the item shown was the required 
piece of test equipment, observe and note any 
errors made in connecting the equipment, observe 
and critique the control settings used by the 
demonstrator, and observe and interpret the 
results obtained. In most cases the tapes for testing 
the use of special test equipment items were 
lengthy since the set of equipment involves 
numerous connections and ancillary equipments. 

The video test materials for the general test 
equipment followed the same format, however, 
this equipment (i.e., voltmeter, oscilloscope, tube 
tester and transistor tester) was tested on non- 
prime equipment in the actual performance tests. 
Special signal generators for the voltmeter and 
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oscilloscope were used to have a complete and 
standardized set of test responses required. The 
same input devices were used in the video test so 
that a comparison of results could be made. In 
addition, however, it was necessary for the test 
subject to interpret results obtained on these items 
of test equipment during the course of the trouble- 
shooting problems. That is, voltage readings and 
oscilloscope displays had to be read and inter- 
preted as part of these problems. The test subjects 
were not evaluated on the basis of that interpre- 
tation, per se, however, in the troubleshooting 
test. Therefore, separate test equipment usage tests 
were developed as a basis for dignosing specific 
individual skill areas. 

Alignment and Adjustment Tests, The video 
materials to test this functional area were devel- 
oped on the basis of depicting co 'ect and 
incorrect procedures for accomplishirg specific 
adjustments on the equipment. A specific named 
adjustment was announced as the subject of the 
test. The video material then showed a technician 
proceeding to accomplish that routine. He first 
selected the equipment he would need, made the 
equipment interconnections, demonstrated that 
the subject test value is out of tolerance, manipu- 
lated the controls on the prime equipment that 
affected that parameter (e.g., coils, trim pots, etc.) 
and then rechecked the output values. The test 
subject observed the actions of the technician- 
demonstrator and was asked specific questions 
about the events on the screen as the action 
progresses. He was required to indicate whether 
proper equipment connections were made, 
whether tolerance values were correct, whether 
correct controls were adjusted, and whether the 
equipment was property adjusted. 

General Tool Usage, In electronics maintenance 
the most commonly used hand tools and the most 
critical are those associated with soldering. The 
tool usage test, therefore concent r es on this skill. 
The other hand tools employed (screwdriver, 
wrenches, etc.) are represented in the removal and 
replacement task and are considered too rudi- 
mentary to be covered in separate usage tests. 
Soldering, however, is a complex skill made up of 
many separate sub-skills. The individual must have 
physical dexterity and be able to locate the proper 
components, select proper replacements, select 
proper soldering tools, remove components 
without heat damage to adjacent components, and 
install a replacement property, again without heat 
damage. 

The video tape materials for this test consisted 
of two parts. The first presented the array of 
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soldering tools and quizzed the subject on their 
proper employment. The second part showed a 
series of piece/part removals and replacements 
illustrating common errors in tool selection, 
installation and component selection. The test 
subject is to observe the procedure used and 
answer specific questions about what was done 
correctly and incorrectly. 

Troubleshooting. The video tests for trouble- 
shooting incorporate all of the previously covered 
skills into an integrated performance directed at 
finding and correcting a system malfunction. In 
the performance tests of troubleshooting the 
individual was simply told that the radar set was 
inoperative and he was to fix it. This meant he had 
to start at the beginning of the checkout routine 
and track down through the system to the defective 
part. He was permitted to use any techniques or 
procedures he chose without penalty, being graded 
only on whether he found the problem. 

The video materials had to incorporate some 
troubleshooting strategy, however. Therefore, the 
test subject was shown the demonstrator carrying 
an action to a decision point and then the test 
subject was queried about his interpretation and 
decision based on what he had seen. He was then 
shown what the demonstrator had decided to do 
which could e»*her be right, wrong or neutral (i.e., 
different but not incorrect). Whenever the test 
subject felt he had enough information to reach a 
conclusion about the fault he could indicate his 
answer. The video tape material proceeded to 
show the demonstrators actions as he made his 
decisions and attempted various corrective actions. 
After each attempted corrective action it would be 
checked out and the test subject would indicate 
whether or not the fault had been corrected, and, 
if not, what should be done next. 



ni. RESULTS 

Test Material Evaluation 

The video materials developed in this project 
were subjected to several forms of evaluation at 
various stages in the project. These consisted of 
evaluations by psychologists viewing the materials 
and critiquing the test strategies employed and by 
administering the^test materials to both novice and 
experienced electronic technicians. These test 
administrations, however, never advanced to the 
point of giving both the video and actual per- 
formance tests to any of the technicians for 
comparison purposes. This was due to the diffi- 
culties encountered in getting a satisfactory 



version of the video tests to warrant such a 
comparison. The results reported herein, then, are 
what was found in administering just the video 
materials, observing and questioning the test 
subjects on their reactions to the materials, and 
their relative success in the tests themselves. 

General Test Equipment Tests (GE), The 
following items of equipment made up the GE 
category of tests: 

• VTVM 

• Oscilloscope 

• TV-2 Tube Check 

• 1890M Transistor Tester 

In transitioning the general equipment usage 
tests to the video format, the hypothesis was that 
the added fidelity of the displays, with actual 
meter movement where appropriate, would result 
in more valid tests or equally valid but more 
interesting tests. The actual result, however, was 
that the video versions of these tests remove at 
least one degree of freedom from the graphic 
versions - that of allowing the subject to adjust 
the control settings - while not significantly 
improving the display fidelity. Neither were 
advantages of interest or production efficiency 
achieved. As a matter of fact, both were at a 
disadvantage when compared with the graphic 
versions. Since video must be paced for the slowest 
individual in a group, it tends to be boring for the 
average and faster subjects. In terms of pro- 
duction, the cost in time and labor is much higher 
than for the graphic versions, so no advantages are 
gained in that respect. 

The most serious disadvantage found with the 
uye of video to measure test equipment usage 
abflity was the need for lock -step presentation of 
information.. Typically, each individual will 
examine the piece of test equ^ment in a random 
and unique fashion, therefore, he will operate best 
when having a fixed, statio display (either real or 
simulated), given that he can inspect, consider, and 
reconsider, using the display as an aid to recall. A 
video display, due to limitations of lenses, must 
focus on one control position at a time. The 
subject has to accept or reject the relevance of that 
control and its value setting when it is siiown. He 
cannot come back to it at will. Thus, the indi- 
vidual with less than highly-developed usage skills 
is penalized by lack of observation control while 
the proficient individual must view much more 
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information than he needs. Since the video testing 
techniques for general equipment usage hd\e 
multiple disadvantages while offering virtual)) no 
advantages, it is concluded that performance on 
this type of task is not best tested via the video 
medium. 

Removal and Replacement Tests. The primary 
limitation in testing these types of tasks via video 
is in the lack of wrong alternative procedures that 
can be presenK^d. About ail that can be shown 
incorrectly is the removal of the wrong element, 
which is primarily testing component identifi- 
cation skills. This skill is readily tested with 
printed graphic materials. The ele:nents of the 
removal and replacement task for electronics are 
basically very simple once location is known. For 
the most part, they consist of manipulating screw 
fasteners and plug-in units, such as tubes and 
circuit boards. Therefore, the physical demands 
are limited, and, due to the size of the equipment, 
there are few critical sequential aspects that can be 
portrayed. Video testing of these tasks has not 
been found to offer any advantages over the 
graphic symbolic media. 

Special Test Equipment Usage Tests, The only 
way to employ the items of test equipment 
covered in this category (UPM-10, Signal Gener- 
ator, URM-25, and Audio Oscillator) is in the 
process of performing a specific alignment or 
adjustment on the prime radar equipment. The 
video materials developed for the tests on these 
items of special test equipment, then, overlap with 
that portion of the larger alignment/adjustment 
test. They duplicate that portion dealing with the 
use of the particular item of test equipment, but 
are presented and evaluated differently. 

In the test equipment usage test, the elements 
of the task are evaluated in greater detail than in 
the adjustment task. The elements of this task that 
are evaluated are identification of the test 
equipment, connecting the test equipment for use, 
correct manipulation and setting of the controls, 
both on the priiiic and test equipment, and reading 
and interpreting results. 

While the degree of overlap of these tests with 
the associated adjustment tests varied, in most 
cases it constituted a significant and critical 
portion of the adjustment task. This is particularly 
true of the video tests, more so than the perform- 
ance tests. In performance tests of adjustments, 
the individual has to demonstrate that a radar set 
has been put back into tolerance on the subject 
parameters. This is all the Test Administrator has 
to grade. The test subject can use any equipment 



he wants, although there are not generally 
workable alternatives available. In the video 
version of the adjustment task, however, since the 
fin^ result cannot be controlled by the test 
subject, must evaluate processes in order for the 
examiner to know whether he has an under- 
standing of the correct procedures and outcomes 
soijht. 

In the performance test for test equipment 
usage, the test subject is given the same general 
assignment as in the corresponding adjustment 
task only he is instructed that he must use the 
particular item of special test equipment under 
test, even if he would normally employ other- 
equipment or techniques in its place. In the per- 
formance tests, there was the possibility of a 
difference between the way the same task was 
performed in the above two tests and the grading 
sy^em was set up accordingly. In the video 
version, however, the two were by definition the 
same, since it was considered legitimate only to 
present approved methods of task performance 
with the labe' "correct" on them. Local variations, 
procedures, and short-cuts were tolerable in the 
performance tests but could not be promulgated 
via the video tests. Due to this situation, then, the 
evaluation of special test equipment usage capa- 
bility was integrated into the appropriate align- 
ment/adjustment test. 

System Checkout Tests. A somewhat similar 
situation existed with the system checkout 
procedures. The basic connections necessary to 
make a radar system checkout are lengthy and 
involved, so that in actual practice a "bench set- 
up", once established, is maintained intact even 
though it may be used only occasionally. The 
checkout portion of the task, however, is 
employed repeatedly as the initial part of the 
troubleshooting process. The bench set-up capa- 
bility was tested once in the performance tests as a 
specific skill, but was not required each time a 
checkout was made. In the video tests a com- 
parable division of the task was made. The check- 
out was tested separately in the performance test to 
provide a separate measure of that skill since, 
while it is part of the troubleshooting process, it 
was not separately evaluated. In the video tests, 
however, it is measurable each time a trouble- 
shooting test is taken, since the checkout process 
is performed in standard fashion each time. 
Therefore, a separate test of checkout procedures 
was redundant in the video tests, whereas it was 
necessary in the performance tests. 
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General Tool Usage Tests, The ability to solder 
properly is a skUl that has thus far defied satis- 
factory testing via symbolic techniques. It is a 
combination of technique and individual physical 
dexterity and only the technique portion is 
ultimately testable in symbolic format. Recog- 
nizing this limitation, graphic symbolic tests were 
developed which dealt with the individual abflity 
to recognize proper soldering jobs, to identify 
correct and incorrect part replacements, and to 
translate from schematics to actual parts location. 
All of these were considered as ancillary soldering 
skills. To test these skills, however, it was 
necessary to segment them considerably. Judging 
completed solder joints was one segment, selecting 
proper tools another, and correct parts installation 
a third. 

in the first area, video did not offer any signif- 
icant advantage since sufficient resolution could 
not be achieved to permit accurate discrimination 
of good and bad solder joints. In the latter two 
areas, however, video does permit a much more 
realistic problem to be formed. The soldering 
process can be shown combining the elements of 
equipment usage and parts replacement in a 
manner closelv resembling the actual job. A 
number of questions can be asked concerning 
these aspects of the job as they are being per- 
formed on the screen, which is more realistic than 
the graphic exercise which requires interpretation 
of a representational line drawing. Thus, while the 
evaluation of physical dexterity related to 
soldering is still not measured, the related skills 
measurement appear^! to be improved with the 
controlled video presentation. 

Alignment/Adjustment Tests, Alignments and 
adjustments on electronic gear involve the use of 
numerous items of test equipment and specific 
processes to check and set the \arious operational 
parameters of the equipment. The elements of the 
task are: knowledge of appropriate test equip- 
ment, proper hook-up of equipment; checkout 
procedures; location of adjustment control; 
application of tolerance limits, and interpretation 
ot readings so that proper adjustment is made. 

In converting to graphic symbolics for these 
tests, the elements retained were equipment 
connections (drawing lines between equipments), 
location of adjustment control (by marking 
pictures), identification of test equipment to be 
used, and results interpretation Cby interpreting 
display pictures). The evaluation of the subject's 
performance is made in terms of the correctness of 
each of his responses to these test items, since the 



test materials are not responsive to his actions. The 
evaluation scheme shifts, then, from the product- 
oriented type to the process-oriented type. 

In applying video technique to the alignment/ 
adjustment test area, attempts were made to 
utilize the dynamic aspects of video to increase the 
realism of the task. A demonstration of the 
specific alignment procedure was staged to illus- 
trate both correct and incorrect procedures being 
used at various points in the performance. During 
the showing of the tape, subjects are quizzed 
about what is being done. They must "critique" 
the demonstrated performance by answering 
questions about it. The topics covered are equip- 
ment set up, results interpretation, tolerance limits, 
and test equipment usage. 

While video offers some advantages in the 
testing of alignment and adjustment capabilities 
(in terms of displaying for the test subject the 
myriad of connections to be considered and the 
dynamic response of the output indicators), this is 
one of the areas where integration of documen- 
tation usage is most difficult and important. These 
procedures are seldom performed without 
constant reference to the technical documen- 
tation. The technician uses it for checking on 
procedure and confirming tolerances. When 
attending to the video presentation and verifying 
the procedures being used therein, it is impossible 
to shift attention to the documentation unless the 
videotape is stopped to permit this. In addition, 
the lengthy checkout routines become even longer 
when each step has to be carefully and distinctly 
demonstrated. 

Troubleshooting Tests. The tryout of the 
troubleshooting materials indicated the difficulty 
of having to impose an arbitrary strategy on a 
technician and have him evaluate it. It appears that 
one of the critical links required to do this is a 
description of the rationale being employed by the 
demonstrator. That is, when the technician taking 
the test sees a symptom or malfunction indication, 
he may make assumptions based upon logic, 
experience, or hunch any of which may lead to a 
different successive action. When the demonstrator 
in the film takes a follow-up step it may or may 
not correspond to that which the test subject 
would have taken. Whether either is "incorrect" is 
a moot point. In terms of the performaiKe tests 
the procedures taken would not have been eval- 
uated per se, only whether or not the malfunction 
was fixed. To have the test subject view a 30 - 45 
minute tape sequence and then describe what was 
done incorrectly was, of course, unreasonable. The 
initial approach used was to have him viev/ each 
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discreet operation, (e.g., system checkout, power 
supply check, etc.) and then answer several 
questions about the procedures used, the results 
obtained, and the recommended next action. The 
test subjects became bored even when the tape was 
broken into viewing and testing segments of S-10 
minutes. It was simply tedious to sit and watch a 
very methodical depiction of all the actions. The 
actions had to be methodically portrayed, of 
course, otherwise there was no way to make sense 
out of them. One had to know precisely which test 
point was being checked or which trim pot was 
being adjusted to evaluate what was happening. To 
display such information required careful camera 
positioning and then, because of the cramped 
location of such items and their similarity in 
appearance, it was necessary to provide the viewer 
with sufficient overview so that he could identify 
exactly where the demonstrater was working. The 
amount of such referencing data that was 
necessary for the viewer varied with the level of 
experience. Since no assumption of high expe- 
rience could be made, a high level of reference 
views had to be included. This contributed to the 
slow pace of the action on the fBm and the length 
of the time required to j^rescnt sufficient infor- 
mation to frame the problem. 

Evaluation of Video as a 
Performance Testing Medium 

Based upon the results obtained in the indi- 
vidual test areas, it was concluded that video has 
several inherent characteristics that makes it 
undesirable as a medium for administering per- 
formance tests in electronics maintenance. These 
characteristics affected all of the individual test 
areas, and were found to outweigii the originally 
sought advantages of prime equipment reduction, 
group administration, and dynamic display 
characteristics. Specific deficiencies identified are 
summerized as follows: 

Presentation Time Required. In order to solve 
problems in electronic maintenance it is necessary 
for the technician to have access to a large amount 
of data about equipment conditions. He must then 
be able to select the re^ vant and necessary data 
from that information. If the symptoms of a 
malfunctioning radar are *o be interpreted, for 
example, it is necessary for the student to sec how 
they were obtained, where any readings or 
measurements were taken and what the exact 
results were. To portray this in meaningful context 
via video takes about as long as for the technician 
to do it himself, less, of course, the normal time 
required to find equipment, gather up tools, 



connectors, etc., and the other time consuming 
side activities associated with performance in a 
shop environment. While actually performing the 
test tasks keeps the subject occupied and perhaps 
interested, watching someone else do it does not. 
While all that is needed is to know what the results 
of such checks are, results given out of context are 
confusing. Further, to boil down results to a series 
of displays removes the main function of video, 
which is to show the details of process. 

Passive Nature of Tests. Video requires the test 
subject to observe and react to the actions shown 
on the screen, and he cannot control or alter what 
is presented. Since the video must be presented in 
afixed sequence, that sequence must be prescribed. 
Th e information on tape cannot readily be 
randomly accessed with present equipment. There- 
fore, it cannot give him a resuh based upon his 
action. The video suffers, therefore, from the lack 
of responsiveness just as do the graphic symbolic 
forms of the test. 

The passiveness of the video, combined with 
the extended presentation time required, creates a 
problem in that it is boring to watch someone else 
perform for an extended period. Yet the observer 
must remain attentive for if he misses a critical 
item, what follows can become meaningless. This 
is not as true when the subject is performing 
himself. He may make a mistake but he goes on 
with hypotheses and beliefs that are meaningful to 
him. That is, he may be wrong, he may je per- 
forming meaningless steps so .far as the "real*' 
problem is concerned. But if he doesn't know that, 
it has some meaning for him • even though it*s the 
wrong meaning. The point is that he doesn't 
become bored or distracted so long as it seems 
meaningful to him. 

Conflict with Documentation Usage. Since 
video has, by definition, high visual demands, it 
presented a basic conflict with a dominant visual 
demand of electronics maintenance for the use of 
documentation. To be realistic, the problems 
presented in the perfomiance tests require that the 
subject be permitted to refer to his technical 
documentation. The technician must use such 
information on the job and he is not supposed to 
rely on his memory for technical data. In video 
testing, however, the situation is encountered 
where the subject must attend to two information 
sources -the video and the technical documen- 
tation. To produce video tests that can be 
validated against performance tests, the use of 
such documentation must be an integral part of 
the test design. This introduces the need to predict 
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both the points at which the subjects will have to 
refer to their documentation and the amount of 
time that will be required. Such requirements will, 
of course, vary with individuals and allowance has 
to be made for slower individuals. Alternative 
methods for accomplishing this are available based 
upon the administrative techniques emi)loyed. If 
the tests are to be strictly standardized in terms of 
administration, then the frequency and duration 
of time allowed for documentation usage can be 
controlled by the voice-over on the tape, along 
with the regular narration. Alternatively, if less 
control is possible, then the Test Administrator 
could control the use of the documentation. In 
this approach the Test Administrator would 
observe the subjects and when he judged that 
sufficient time for documentation usage had been 
allowed, he would re-start the video tape. Doing 
this would mean that the frequency and time 
allowed for documentation usage v/ould vary from 
administration to administration. If this variation 
can be tolerated it would permit the test to 
continue when everyone is ready to continue, 
rather than waiting for a fixed interval. 

The question of documentation usage is further 
complicated when the video tests are administered 
to personnel using Proceduralized Job Perform- 
ance Aids (Foley, 1973). Using such aids, the 
subjects are to perform every maintenance action 
according to specific written instructions. Theo- 
retically all decisions about what actions to take 
have been accounted for in these instructions. Use 
of such materials has two major impacts on video 
tests. First, time for documentation reference has 
to be provided more frequently. The test subject 
would have to check each step against the 
documentation to know whether it had been 
performed correctly since much less information is 
supposed to be in the JPA user's head than for 
conventionally trained personnel. Second, 
questioning the subject as to the most appropriate 
ne);t step would not be meaningful since the next 
rt&p is directed by the JPA. JPA users do not 
generate their own troubleshooting strategy. 

Video Material Development Costs, A major 
consideration with respect to the use of video test 
materials is the expense involved in their produc- 
tion. This expense is incurred in several ways. 
Equipment is required that will produce an 
acceptable product and facilities are required for 
the production of the materials. These will likely 
be operational facilities rather than studios since 
that is where the equipment and personnel are 
located. Working in these environments is less than 



maximally efficient since operational priorities can 
intrude at any time. Further, videotaping requires 
sufficient control of conditions so that usable 
material will be obtained, which involves setting 
up shots, controlling the actions of the demon- 
strator, and insuring minimum essential environ- 
mental conditions. In doing those things, expense 
is incurred either in video equipment and 
accessories or in increased time and personnel 
costs to compensate for equipment shortcomings. 
In the latter case limits of such compensation are 
reached where increased production time cannot 
make up for equipment deficiencies. 

The approach taken in this study was to start 
with limited equipment on an experimental basis 
to determine whether a greater investment was 
warranted. In some respects, this guarantees 
marginal results. To obtain video pictures with 
sufficient resolution for any degree of detailed 
observation, it is necessary to have good equip- 
ment at each critical point in the system - camera, 
recorder, and monitor. Next there must be a 
minimum of two cameras (and probably more) to 
provide proper viewing angles, and this requires 
some switching equipment, as Well as muhiple 
monitors for the cameras. An adequate assorts ent 
of lenses is required to provide flexibility, and 
supplemental lighting will likely be required. If 
any editing of tape is required (and it almost 
certainly will be), then a second recorder com- 
patible with the first must be available. In this 
project, with one camera, recorder and monitor, a 
minimum of two people were required to operate 
the equipment and control the action of the tech- 
nician-demonstrator. The recording was, of course, 
conducted in a nonstudio environment. However, 
it seems that this would very often be the case, if 
one were dealing with operational prime equip- 
ment that could not be readily made available in a 

StU(liO. 

As the basic system and operational conditions 
get more complex, the requirements for operator 
personnel may increase, with resulting increase in 
costs. In this project, approximate!/ 15 hours of 
video tape material was recorded. Over 700 man- 
hours were required to obtain this material. While 
this included some excess time as various alter- 
natives were tried out, and certain other experi- 
menting, it also included down time spent while 
repairing the prime radar equipment, yielding to 
higher priority work requirements at the site, and 
waiting for test equipment to become available. 
These are typical costs that hnve to be paid when 
the video taping mission is secondary. TTiis, too, is 
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a high probability condition. The total time 
required could likely have been reduced by more 
equipment. As it was, considerable time was 
required to shift the camera, lights, and human 
subject when a different view was necessary. If 
pre-positioned cameras were installed, then such 
shifts could be accomplished by a switching 
device. 

Once the desired material is on tape the next 
step is to edit it. This consists of selecting the 
specific sequences desired, eliminating unwanted 
material and adding the desired narration. This 
process is not easily accomplished and can best be 
facilitated by having a complete and accurate log 
of the location of each specific action sequence. 
The desired final tape should then be programmed 
on paper first to define the desired sequence such 
as spacing and tape source. This will greatly facfli- 
tate the actual manipulations of the videotape, 
which is desirable since the resolution quality and 
durabflity of the tape decrease rapidly with 
frequent stopping, starting, and re-winding. 

To accomplish the editing functions requires a 
second recorder so that tape material can be 
transferred onto a second tape at the desired 
location. This process requires equipment with 
electronic editing capability tc produce satis- 
factory results. Without it, the junctures at which 
materials are inserted produce considerable visual 
noise caused by the start up of the recorder 
mechanism. 

The value of the equipment used in this project 
was about $2,500. A minimum of equipment 
estimated as required to produce tapes of suffi- 
cient quality for viewir.g and reproduction would 
be $7,500 to $10,000. This would provide two to 
three cameras, assorted lenses, two 1/2-inch video 
tape recorders with electronic editing and remote 
control, three monitors, a switching network, 
auxiliary lighting, tapes, and associated hardware 
for inte reconnect ion. A minimum amount would 
also have co be invested in carts, stands, etc., in 
lieu of a console. This is estimated to comprise a 
minimum system to produce adequate video tapes 
under nonstudio conditions. 



IV.CONCLUSIONS AND RECOMMENDATIONS 

Conclusions 

1. The testing strategies available for video 
application are not very appropriate to the nature 
of the electronic technician's job. They are limited 



in permitting a technician to pursue alternative 
courses of action, cannot conveniently permit use 
and study of technical documentation, and require 
a lock-step presentation that is not efficient for 
this type of work. 

2. For the most part, these testing strategies 
violate a major ground rule of criterion referenced 
tests for electronic maintenance (as discussed in 
Vol II of this series of reports) which places the 
measurement emphasis on an acceptable product. 
In most cases, video test strate^es must place the 
measurement emphasis on process. 

3. Video symbolic tests place the test subject 
in a passive, evaluative role of watching someone 
else perform the job task. Success in this passive 
role does not ensure his success in the active role 
of performing the task. The passive, evaluative role 
is an appropriate one for a job supervisor, a test 
administrator, or an instructor - not for an elec- 
tronic technician whose job is to perform the task. 

4. This effort indicated that the cost of devel- 
opment of video symbolic tests is very high in 
terms of video equipment and test development 
personnel time. 

5. Development of video tests in an operational 
environment presents production problems that 
limit the resolution of the taped information to a 
less than adequate level for testing purposes. 

6. The use of quality video equipment and 
studio production conditions is the only way that 
video materials of acceptable quality can be 
produced for testing purposes. 

7. Due to development and administrative cost 
and the other shortcomings indicated above, the 
administration of these video symbolic tests never 
advanced to the point of giving both the video and 
the job task performance criterion tests to any 
technicians for systematic comparison purposes. 

8. This video effort was an attempt to ''short 
circuit" some of the administrative problems that 
must be solved for graphic type symbolic substi- 
tute tests. The attempt to use video for this 
purpose surfaced expensive developmental and 
administrative problems and sdved few if any of 
the graphic type symbolic substitute test 
problems. 

Recommendations 

Based upon the foregoing conclusions and upon 
the results of the preceding efforts in this program 
reported in the first three volumes of this series of 
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reports, the foBowing recommendations are made 
regarding future development and use of symbolic 
testing materials for electronics maintenance. 

1 . The method of video presentation described 
in this document should not be considered as a 
testing medium for performance analogues. The 
method of presentation described places the test 



subject in an observational rather than a per- 
formance mode. 

2. Future efforts should be aimed at improving 
and reflning graphic symbolic substitute tests in 
keeping with the recommendations presented in 
Volume III of this scries of reports. 
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